Deception Detection Via Blob Motion Pattern Analysis

Deception dectection is one of the most difficult problems in affect recognition and expression research area. Recently, non-verbal methods of detecting deception have appeared to be promising. Thomas[1] presented a proof-of-concept study based on the blob analysis of some suspects’ interviews and mock experiments video clips. In this paper, we present our recent research work in the direction of developing an automated deception detection system. We propose a blob motion pattern analysis approach to solve this problem. Our approach consists of the following steps: (a) using skin-color based technology to detect body blobs, i.e. head and hands, and calculating the in-frame and cross-frame features. (b) segmenting the training videos into small clips with a fixed duration where each clip contains only one blob motion pattern, and automatically clustering these patterns into groups. (c) using HMM-based method to model the pattern sequences and estimating the latent subject’s state. During the body blobs detection process, we first take an off-line training phase to set up a lookup table to determine the probability of each color vector ci being skin-colored. Then a standard connected components labeling algorithm is applied to yield different skin-colored regions. Size filtering on the derived connected components is performed to eliminate small, isolate blobs that do not correspond to body blobs. An ellipse fitting algorithm is also applied to find the right position of body blobs. We calculate the in-frame and cross-frame features include position, shape, distance between blobs, velocity, etc. and denote each blob as a vector b = {position, size, velocity, . . .}. Due to the space-time nature of blob motion patterns, we adopt a discrete scene event based feature representation approach considering that each clip contains only one blob motion pattern. The blob motion pattern in a clip can be represented as P = {f1, f2, . . . , fT}, where T is the number of frames in the video clip and fi = {bhead, bleft−hand, bright−hand}. Now the deception detection problem can be redefined formally. Consider the training data set D consists of N patterns with a fixed duration, D = {P1, P2, . . . , PN}, where Pi is the pattern vector defined as above. The subject’s behavioral state can be considered as a discrete time series of patterns. Using these patterns as observations we can train an HMM to estimate the latent subject’s state. We assume that there are 3 hidden states in the model, denoted as S = {agitation, normal, over − control} This assumption comes from the